Multi-scale computing over heterogeneous resources

Parallel processing of large data sets on clusters of commodity machines—of virtual or physical nature—has in recent years become an active field of research as well as a thriving business. Many companies are running large data centres to support such workloads, and task-parallel frameworks have enabled programmers to harness managed large-scale parallelism while writing mostly sequential code. At the same time, the increase in individual per-core CPU performance has slowed down significantly and the industry has moved towards parallelism as a way of increasing CPU performance. Non-cache-coherent manycore CPUs are under active research and development, and the average commodity server now contains between four and eight logical CPU cores. On top of this, increasing awareness of data centre energy consumption as an environmental and economic challenge has led to diversification of hardware used in compute and storage clusters [1]. The systems used to run such jobs across many machines, however, essentially treat them as a homogeneous collection: they partition them into many VMs at the hypervisor level or coarse-grainedly break them up into a number of “slots” per machine. Nonetheless, heterogeneity in the data centre is a reality at multiple levels—from hardware and interconnects to task-level heterogeneity within a job [5]—and may even be introduced deliberately [2]. We believe that there has been no appropriate general treatment of resource heterogeneity in compute clusters so far. In this work, we introduce the concept of independent resource ensembles, marking a departure from the classic single global master design of taskparallel compute frameworks (§2), and propose a new approach to detecting and characterizing resource heterogeneity within and between ensembles (§3), enabling heterogeneity-aware scheduling decisions. In explicitly supporting heterogeneity-awareness and dynamic extension of a computation across resource ensembles, we introduce the idea of multi-scale computation—that is, we can exploit parallelism at many scales, from fine-grained multi-core processing with communication over an onchip network to coarse-grained clusters of virtual machines (§4).